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First Things First 

staff members and students to monitor and support progress 
and that serves as a bridge bet\ween the school and famiiy, and 
(3) instructional improvements to make classroom teaching 
more rigorous and engaging and more closely aligned with state 
standards and assessments.'' 



Program description First Things First is a reform model intended to transform 

elementary, middle, and high schools serving significant propor- 
tions of economically disadvantaged students. Its three main 
components are: (1) “small learning communities” of students 
and teachers, (2) a family and student advocate system that pairs 



Research 



One study of First Things First met the What Works Clearing- 
house (WWC) evidence standards with reservations. The quasi- 
experimental research design included students from Houston 
high schools— 3 First Things First schools, each matched to 10 
or 11 comparison schools. Based on this one study, the WWC 



considers the extent of evidence for First Things First to be small 
for staying in school. That study did not examine the effective- 
ness of First Things First in the domains of progressing in school 
or completing school. ^ 



Effectiveness First Things First was found to have no discernible effects on staying in school in its first year of implementation. 





Staying in schooi 


Progressing in schooi 


Compieting schooi 


Rating of effectiveness 


No discernible effects 


na 


na 


Improvement index'^ 


Average: -1 percentile point 


na 


na 








na = not applicable 



1. The descriptive information for this program was obtained from publicly available sources: the program’s website ( www.irre.ora/ftf . retrieved October 
2007) and the research literature (Quint, Bloom, Black, & Stephens, 2005). The WWC asks developers to review the program description sections for 
accuracy from their perspective. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2. The evidence in this report is based on available research. Findings and conclusions may change as new research becomes available. 

3. These numbers show the average improvement indices for all findings across the study. 
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Developer and contact 

Information on First Things First is available from the Institute for 
Research and Reform in Education (IRRE), which developed the 
intervention. Address: 308 Glendale Dr., Toms River, NJ 08753. 
Web: www.irre.org . Telephone: (732) 557-0200. 

Scope of use 

First Things First was first implemented in 1998 in the Kansas 
City, KS, school system. IRRE reports that, as of December 
2007, 11 districts in seven states were operating schools using 
the First Things First model. 

Description of intervention 

First Things First is a reform model designed to boost student 
achievement in schools serving a large number of economically 
disadvantaged students. The model’s goal is to help students 
acquire the skills needed to succeed in postsecondary educa- 
tion and the labor market. It has three main components. 

Theme-based small learning communities. First Things First 
reorganizes high schools into small learning communities of 
up to 350 ninth to twelfth graders and their teachers, each with 
a guiding curricular theme (such as science and technology). 
Twelfth graders can participate in internships associated with the 
theme of their learning community. As students progress through 
high school, they remain in their learning communities, with the 
same peer group and teachers. 

Family and student advocate system. Each student in a First 
Things First school is assigned an advocate, typically a teacher 
from the small learning community who serves as a mentor and 



a liaison between the school and the student’s family. Advocates 
work with about 15 students and meet with them weekly in 
groups and one-on-one. They also contact the student’s family 
regularly to discuss academic progress and any challenges fac- 
ing the student in and out of school. 

Instructional changes and supports. First Things First empha- 
sizes a more rigorous and engaging curriculum closely aligned 
with state standards and assessments. The model calls for a 
careful review of all course offerings to ensure that they closely 
correspond to state curriculum standards. It also directs schools 
to develop and regularly administer common assessments that 
reflect these standards and that mirror the format and content of 
state tests. Student performance on these regular assessments 
is then used to guide and improve classroom instruction. First 
Things First also offers professional development and technical 
assistance to improve the rigor of course offerings and the ability 
of teachers to reach students of all learning styles. 

Cost 

According to IRRE, the additional annual cost of operating First 
Things First in the first two years of program implementation 
(above and beyond the cost of traditional high school) ranges 
from $150 to $275 per student. Beyond the initial start-up phase, 
annual per student costs are somewhat lower, ranging from $100 
to $175. These costs include curriculum, materials, and ongoing 
technical assistance. IRRE reports that the per student cost of 
implementing First Things First varies depending on the size and 
number of the schools implementing the model, current staffing 
levels, and other factors. 



Five studies reviewed by the WWC investigated the effectiveness 
of First Things First. Four studies were included in one research 
report (Quint, Bloom, Black, & Stephens, 2005), but of the four, 
only the study in Houston, Texas met WWC evidence standards 
with reservations. The three other studies— conducted in River- 
view Gardens, MO; Kansas City, KS; and Shaw and Greenville, 
MS— did not meet WWC evidence screens. A fifth study of First 



Things First (Gambone, Klem, Summers, Akey, & Sipe, 2004) also 
did not meet WWC evidence screens. 

The Houston study included in the Quint et al. (2005) report 
focused on three Houston high schools that implemented First 
Things First from 2001 to 2004. These three schools were each 
matched to high schools in the district that did not implement 
First Things First but had similar achievement test scores. To 
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Research (continued) 



Effectiveness 



The WWC found First Things 
First to have no discernible 
effects on staying in school 
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estimate the effect of the program the researchers first compared 
the average outcomes of ninth graders who entered First Things 
First high schools in the years immediately after the program 
was implemented with those of ninth graders from the same 
schools in the three years just before program implementation. 
They made similar calculations for the comparison schools. Their 
estimates of the effect of the program represent the difference 
between these pre-post implementation comparisons in First 
Things First high schools and the comparison schools. The study 
made similar estimates of the effects of First Things First in a set 
of Houston middle schools but did not include any outcomes 
relevant to the WWC review of dropout prevention interventions. 



Extent of evidence 

The WWC categorizes the extent of evidence in each domain 
as small or moderate to large (see the What Works Clearing- 
house Extent of Evidence Categorization Schemed The extent 
of evidence takes into account the number of studies and 
the total sample size across studies that met WWC evidence 
standards.'* 

The WWC considers the extent of evidence for First Things 
First to be small for staying in school. No study that met WWC 
evidence standards addressed the domains of progressing in 
school or completing school. 



Findings 

The WWC review of dropout prevention programs addresses 
student outcomes in three key domains: staying in school, pro- 
gressing in school, and completing school. The Houston study 
by Quint et al. (2005) assessed outcomes in the staying in school 
domain. 

Staying in schooi. In the Houston study Quint et al. (2005) found 
no statistically significant difference after one year of implementa- 
tion between First Things First schools and comparison schools 
in the percentage of ninth-grade students who attended school 
the following year. The effect size was not large enough to be 
considered substantively important by WWC standards. 



Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as: positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of 
the difference between participants in the intervention and the 
comparison conditions, and the consistency in findings across 
studies (see the WWC Intervention Rating Scheme) . 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index across studies (see the 
WWC Improvement Index Technical Paper) . The improvement 



index represents the difference between the percentile rank 
of the average student in the intervention condition and the 
percentile rank of the average student in the comparison 
condition. Unlike the rating of effectiveness, the improvement 
index is based entirely on the size of the effect, regardless of 



4. The Extent of Evidence Categorization was deveioped to teii readers how much evidence was used to determine the intervention rating, focusing on the number and 
size of studies. Additionai factors associated with a reiated concept, externai validity— such as students’ demographics and types of settings in which studies took 
piace— are not taken into account for the categorization. Information concerning how the extent of evidence rating was determined for First Things First is presented 
in Appendix A5. 

5. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or 
schools and for multiple comparisons. For an explanation, see the WWC Tutorial on IVlismatch . For the formulas the WWC used to calculate the statistical signifi- 
cance, see Technical Details of WWC-Conducted Computations . For the First Things First report, no corrections for clustering or multiple comparisons were needed. 
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The WWC found First 
Things First to have no 
discernible effects on 
staying in school (continued) 



the statistical significance of the effect, the study design, or the 
analyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable to 
the Intervention group. 

Based on the one study of First Things First that met evidence 
standards, the average improvement index for staying in school 
Is -1 percentile point. 



Summary 

The WWC reviewed five studies of First Things First. One study 
met WWC evidence standards with reservations; the remaining 
studies did not meet WWC evidence screens. Based on this 
one study, the WWC found no discernible effects in the staying 
in school domain one year after program implementation. The 
evidence in this report may change as new research emerges. 



References 



Met WWC standards with reservations 

Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). 
Seating up First Things First: The challenge of scaling up 
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up First Things First: Findings from the first implementation 
year. New York, NY: MDRC. 
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Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). 
Scaling up First Things First: The challenge of scaling up edu- 
cational reform. New York, NY: MDRC.® (Riverview Gardens 
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Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). 
Scaling up First Things First: The challenge of scaling up edu- 
cational reform. New York, NY: MDRC.^ (Kansas City study) 
Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). Scal- 
ing up First Things First: The challenge of scaling up educa- 
tional reform. New York, NY: MDRC.® (Shaw and Greenville 
study) 

Gambone, M. A., Klem, A. M., Summers, J. A., Akey, T. A., & 
Sipe, C. L. (2004). Turning the tide: The achievements of the 
First Things First education reform in the Kansas City, Kansas 
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Strategies, Inc.® 



For more information about specific studies and WWC caicuiations, piease see the WWC First Things First 
Technicai Appendices . 



6. There was only one school in the study condition, so the analysis could not separate the effects of the intervention from the effects of the school. 

7. Unlike the other sites in the MDRC study. First Things First was implemented districtwide in Kansas City. Therefore, unlike the other sites, comparison 
schools were selected from other school districts in the state. The effect of First Things First was estimated by comparing trends in outcomes in Kansas 
City high schools with similar trends for seven other Kansas high schools (outside Kansas City) with substantial minority enrollment and below-average 
test scores. With this methodology, the analysis could not separate the effect of First Things First from the effect of other factors associated with the 
Kansas City school district. 

8. The outcome measures are not relevant to this review. 

9. The study did not use a comparison group. 
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Appendix 



Appendix A1 


Study characteristics: Quint, Bioom, Biack, & Stephens, 2005 — Houston study (quasi-experimentai design) 


Characteristic 


Description 


Study citation 


Quint, J., Bloom, H. S., Black, A. R., & Stephens, L. (2005). Scaling up First Things First: The challenge of scaling up educational reform. New York, NY: MDRC. 


Participants 


The study examined the performance of students attending schools in the Houston Independent School District. The main analysis sample included students from three high 
schools and four middle schools implementing First Things First, as well as students from a set of matched comparison schools. The middle school study did not examine 
outcomes relevant to WWC dropout prevention reviews. Therefore, the results in this intervention report pertain only to the high school analysis. 

The high school sample consists of all students enrolled in the study schools during the study period: three years prior to First Things F//sf implementation and one year after 
implementation. This included a total of 7,891 high school studenfs in the First Things F/rst schools. The study authors did not report the number of students in comparison 
high schools. Comparison schools in the Houston school district were matched to each First Things First high school based on overall student performance at the schools on 
standardized achievement tests during the baseline period. High schools in the district whose mean combined reading and math scores on the Stanford Achievement Test, 
ninth edition (SAT-9), were within 0.25 standard deviations of the mean for the First Things First school were selected as comparison schools. This methodology resulted in 10 
comparison schools for 2 of the First Things First high schools and 11 comparison schools for the other. A high school could be chosen as the comparison school for more than 
one First Things First school, and this often occurred. A total of 13 Houston high schools served as comparison schools for fhe First Things F/rsf Houston study. 

The high school study examined two cohorts of studenfs. Cohort 1 consisted of one infervention high school thaf implemented First Things First lo 2001 and its matched 
comparison schools. Cohort 2 consisted of two intervention high schools that began implementing First Things First in 2002 and their comparison schools. The WWC used 
results for cohorts 1 and 2 combined to rate the effectiveness of First Things First 

In the years prior to the implementation of First Things First the three First Things First schools and their comparison high schools had, on average, similar attendance and 
promotion rates and served students who had similar tenth-grade passing rates on math and reading tests. The study authors indicated that the First Things First high schools 
in Houston were heavily Hispanic, while other low-achieving high schools in Houston were heavily African-American. Therefore, it was not possible to match high schools 
closely on both their racial and ethnic composition and their student performance. 


Setting 


First Things Firstms implemenfed in fhree high schools in fhe Houston Independent School District: Lee High School, Sam Houston High School, and Sharpstown High School. 
During the study period, these three high schools served student populations that were primarily Hispanic. Lee High School began implementing First Things First In the fall of 
the 2001/02 school year, and Sam Houston and Sharpstown began implementing the program the following school year, 2002/03. 

(continued^ 
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Appendix A1 Study characteristics: Quint, Bioom, Biack, & Stephens, 2005 — Houston study (quasi-experimentai design) (continued) 



Characteristic 


Description 


Intervention condition 


First Things F/ref targeted three high schools in Houston with low-achievement levels. Each of these schools implemented the key features of the three components of the 
whole school reform: 

Smali learning communities. First Things First reorQan'aes schools into small learning communities of up to 350 students and their teachers. The small learning communities 
in Houston were focused around a central theme (for example, technology), and students remained in the same communities throughout high school. First Things First also 
recommends that schools reduce student-teacher ratios in math and language arts classes to increase the amount of individualized attention that students receive. Class sizes 
in the Houston study schools were reduced from an average of 26 students to 20 students between the planning year and the second implementation year. 

Famiiy and student advocate system. Advocates met one-on-one with students weekly and contacted the students’ families at least monthly to discuss their academic and 
personal progress. In at least one Houston high school the advocates became counselors for the students, helping them to schedule the classes they needed to progress 
toward graduation. Consistent with the model’s guidelines, advocates in Houston high schools met weekly with their students as a group. These group sessions were held as 
either a daily “homeroom” period or a weekly class meeting. 

Instructionai changes and supports. First Things F/rsf contracted with Kagan Cooperative Learning, Inc., to train teachers on cooperative learning strategies intended to comple- 
ment the small learning communities. It also called for increased instructional time for math and language arts courses. 


Comparison 


Matched comparison schools were Houston high schools that did not implement First Things First The study identified comparison schools from the districts that were similar 
in average performance on standardized achievement tests in the three years preceding program implementation. High schools in the district whose mean scores on the 
Stanford Achievement Test, ninth edition (SAT-9), for reading and math combined were within 0.25 standard deviations of the mean for the First Things First school were 
selected as comparison schools. 


Primary outcomes 
and measurement 


The outcome of interest from this study was a measure of the percentage of ninth graders who attended at least one day of school during the following year, referred to as 
the “one-year persistence rate.” The study also examined First Things First’s effects on attendance and standardized test scores. These outcomes do not fall within the three 
domains examined by the WWC’s review of dropout prevention interventions (staying in school, progressing in school, completing school) and are not included in this report. 


Staff training 


Teachers at First Things First high schools were regular teachers employed by the Houston Independent School District. As part of the instructional changes, teachers in First 
Things First schools were trained in cooperative learning strategies by Kagan Cooperative Learning. Each school employed a First Things First director to serve as a liaison 
between the district and the First Things First schools. Staff from IRRE provided ongoing professional development for teachers in the implementation of the family advocate 
system. 
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Appendix A2 Outcome measures in the staying in schooi domain 



Outcome measure 


Description 


One-year persistence rate 


The percentage of ninth-graders who attended at least one day at a district school at any point during the following school year. Students who were recorded in district records 
as having transferred to another district at the end of the ninth grade are coded as having persisted in school. 
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Appendix A3 Summary of study findings inciuded in the rating for the staying in schooi domain^ 









Authors’ findings from the study 












Mean outcome 


WWC calculations 












Statistical 






Study 


Sample size 


First Things First Comparison 


significance^ 


Improvement 


Outcome measure 


sample 


(schools)^ 


group groups Mean difference* 


Effect size® (at a = 0.05) 


index^ 





Quint, Bloom, Black, & Stephens, 2005 — Houston study (quasi-experimental design)® 






One-year persistence rate Ninth graders 


16 76.3 77.3 -1.0 -0.03 


ns 


-1 


(cohorts 1 and 2) 








Domain average for staying in school® 


-0.03 


ns 


-1 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the improvement index. The one-year persistence rate was measured during the second year of implementation of First Things First \n Houston. 

2. Quint, Bloom, Black, & Stephens (2005) used individual student data for their analysis; however, the relevant outcomes are school-level measures. In the study, each block of schools consists of a First Things First schoo\ matched with a 
group of 10 or 11 comparison schools. A school could serve as the comparison school for more than one First Things First school The WWC confirmed with the study authors that the total number of unique comparison schools was 13. 

3. Quint, Bloom, Black, & Stephens (2005) reported only baseline persistence rates and the difference between the baseline to follow-up changes of the intervention and comparison groups. The study authors provided the WWC with 
the follow-up means for both groups. The WWC generated the adjusted comparison group means reported here using the following transformation: adjusted comparison group mean = follow-up comparison group mean -i- (baseline 
intervention group mean - baseline comparison group). Stated differently, the adjusted comparison group mean equals the follow-up intervention group mean minus the impact, since, under the comparative interrupted time-series 
technique used in the Quint, Bloom, Black, & Stephens (2005) study, impacts are calculated as: impact = (follow-up intervention group mean - baseline intervention group mean) - (follow-up comparison group mean - baseline com- 
parison group mean). 

4. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

5. For an explanation of effect size calculation, please see the Technical Details of WWC-Conducted Computations . The effect size for the dichotomous variable “one-year persistence rate” was computed using the Cox Index. 

6. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between groups. The level of statistical significance was reported by the study authors. 

7. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and the average student in the comparison condition. The improvement index can take on values be- 
tween -50 and -i-50, with positive numbers denoting results favorable to the intervention group. 

8. The level of statistical significance was reported by the study authors, or where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation, see the WWC 
Tutorial on Mismatch . For the formulas the WWC used to calculate the statistical significance, see Technical Details of WWC-Conducted Computations . In the case of the Houston First Things f/ref study, the study authors provided 
details of their two-level analysis model, which adjusted for clustering within the school, and thus no additional corrections for clustering were necessary. 

9. This row provides the study average, which in this instance, is the same as the single measure for the persistence rate. 
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Appendix A4 First Things First rating for the staying in schooi domain 



The WWC rates an intervention’s effects for a given outcome domain as positive, potentiaily positive, mixed, no discernibie effects, potentialiy negative, or negative.'' 

For the outcome domain of staying in school, the WWC rated First Things First as having no discernibie effects. It did not meet the criteria for positive effects, poten- 
tiaily positive effects, mixed effects, potentialiy negative effects, or negative effects because it had only one study, and that study showed no statistically significant or 
substantively important outcomes, either positive or negative, in this domain. 

For explanations of how First Things First fared on the criteria for these ratings, see below. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1: No studies showing a statistically significant or substantively important effect, either pos/f;Ve or negative. 

Met. The single study of First Things First showed no statistically significant or substantively important effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No study of First Things First showed a statistically significant positive effect in this domain. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. No study of First Things First showed statistically significant or substantively important negative effects in this domain. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect. 

Not met. No study of First Things First showed a statistically significant or substantively important positive effect in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect and fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. No study of First Things First showed statistically significant or substantively important negative effects in this domain. 



(continued) 
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Appendix A4 First Things First rating for the staying in schooi domain (continued) 



Mixed effects: Evidence of inconsistent effects as demonstrated through either of the following criteria. 

• Criterion 1: At least one study showing a statistically significant or substantively important positive effect, and at least one study showing a statistically significant 
or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. No study of First Things First showed a statistically significant or substantively important effect, either positive or negative, in this 
domain. 

OR 

• Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing a 
statistically significant or substantively important effect. 

Not met. No study of First Things First showed a statistically significant or substantively important effect in this domain. 

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1; At least one study showing a statistically significant or substantively important negative effect. 

Not met. No study of First Things First showed a statistically significant or substantively important negative effect in this domain. 

AND 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, or more studies showing statistically significant or substantively 
important negative effects than showing statistically significant or substantively important pos/'f/Ve effects. 

Met. No study of First Things First showed statistically significant or substantively important positive effects in this domain. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. No study of First Things First showed a statistically significant negative effect in this domain. 

AND 

• Criterion 2: No studies showing statistically significant or substantively important positive effects. 

Met. No study of First Things First showed statistically significant or substantively important positive effects in this domain. 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain-level effect. The WWC also considers the size of the domain-level effect for ratings of 
potentially positive or potentially negative effects. For a complete description, see the WWC Intervention Rating Scheme . 



WWC Intervention Report First Things First 



January 24, 2008 



10 










Appendix A5 Extent of evidence by domain 



Outcome domain 


Number of studies 


Schoois 


Sampie size 

Students 


Extent of evidence^ 


Staying in school 


1 


16 


nr 


small 


Progressing in school 


0 


na 


na 


na 


Completing school 


0 


na 


na 


na 



na = not applicable or not studied 
nr = not reported 

1 . A rating of “moderate to iarge” requires at ieast two studies and two schoois across studies in one domain, and a totai sample size across studies of at least 350 students or 14 classrooms. 
Otherwise, the rating is “small.” 
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